Can one of the admins verify this patch?

@metacosm @attilapiros Some additional thoughts - it seems to be a pretty advanced case to consider using the limit option with an informer. There are quite a few comments around the behavior in the go client.

I don't see much in the literature about it being a built-in case, but it seems like you'd also look towards a sharding pattern - an overall/coordinating operator would create something like a statefulset of controllers, where each has responsibility for a subset of the resources - driven by an annotation (something that is filterable by informers/kubernetes) added by the overall operator.

https://github.com/kubernetes/kubernetes/blob/6f896dec4f45ffb82491bc9ce2393e7452886562/staging/src/k8s.io/client-go/tools/cache/reflector.go#L92

I don't see much in the literature about it being a built-in case, but it seems like you'd also look towards a sharding pattern - an overall/coordinating operator would create something like a statefulset of controllers, where each has responsibility for a subset of the resources - driven by an annotation (something that is filterable by informers/kubernetes) added by the overall operator.

I can't find now the source, but I recall that operators should be singletons (or work with some sort of leader-election mechanism). (Which is something that disappointment given the nature of K8s and its overall purpose)

Is this new sharding approach something documented?

I understand that this has nothing to do with the scope of the current PR, or does it?

Is this new sharding approach something documented?

No, just a musing on how you would approach it assuming a "singleton" coordinating operator.

I understand that this has nothing to do with the scope of the current PR, or does it?

So far the only request for this limit/batching logic is to work towards scaling operators. If there are other applicable approaches we should determine what is best to ultimately peruse. The next step from here is to control the memory footprint of the cache - from #3636 that either looks like a mapping/pruning function or a store that's not fully in memory. I don't think the latter seems like the right direction.

As for this change it does seem like it can stand alone based upon the go client. There are situations where it is beneficial to form smaller paginated results.

No, just a musing on how you would approach it assuming a "singleton" coordinating operator.

To elaborate further - you'd have the coordinating operator set a hash on the resource as a label in the range of the number of statefulset replicas. The statefulset pods would be setup to provide the pod name via an env property, which means they could parse out the pod index to determine which bucket they were responsible for. The individual informers would then be setup to only look for resources with that index label. Any dependent resources would need to have that label transitively set as well for the sharding pattern to work all the way down.

To elaborate further [...]

Yes, I figured some mechanism like this. However, I might still see some flaws depending on the use-case (from my undetailed, first-glance perspective). For some other uses, it might not be necessary to have an orchestrating process/controller (i.e. considering a StatefulSet, the Pod's ordinal name might be enough to devise a constant strategy to self-assign the bucket).

Anyway, I guess this is really off topic for the PR.

the Pod's ordinal name might be enough to devise a constant strategy to self-assign the bucket

To self-assign the hash/index label would already need to be present on the resources.

Anyway, I guess this is really off topic for the PR.

Yes, it was just to add some more to the discussion about operator scaling. I'll add a reference to this to the other on-going issue.

the hash/index label would already need to be present on the resources

Well, in my imaginary example the hash (or input for the resource assignment) could be computed from the resource UID or something similar.

Well, in my imaginary example the hash (or input for the resource assignment) could be computed from the resource UID or something similar.

Something (which I'm referring to as the overall operator) has to compute a hash to the index and apply it as a label to the resource - it needs to be filterable by an informer. I tried to summarize all the extra work this would entail at #3587 (comment)

Well, in my imaginary example the hash (or input for the resource assignment) could be computed from the resource UID or something similar.

Something (which I'm referring to as the overall operator) has to compute a hash to the index and apply it as a label to the resource - it needs to be filterable by an informer. I tried to summarize all the extra work this would entail at #3587 (comment)

😅 I forgot about the filtering part :)

From your detailed explanations I get the following:

• The Orchestrating Pod (Overall Operator) is responsible to assign a filterable property (in this case a label) to the resource to be processed. bottleneck
• 1-N worker Pods with shared informers that filter the assigned resources and perform the actual work. I guess that scaling might impose a challenge in order to rehash the existing resources(I think you expand on this on your comment).

Support for some sort of server-side dynamic labels or wildcard filtering would be of help to completely remove the orchestrating part (kubernetes/kubernetes#30755)

SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

62.8% Coverage
0.0% Duplication

Just wanted to say thanks for adding this feature! My team just ran into needing the ListerWatcher to paginate in the Informer's Reflector and we realized the ability was something we just got from upgrading versions 🙌

Just wanted to say thanks for adding this feature!

No problem, hope it works out well.